Head tracking based gesture control techniques for head mounted displays

ABSTRACT

A head gesture-based recognition system in Headset Computers (HSC) is disclosed. Notification dialogue boxes can be acknowledged by head nodding or ticking movement in the user interface. Question dialog boxes can be assured by head nods or head shakes in the user interface. Head swiping is also a recognizable form of user input through a head tracker of the HSC. Progress indicators and other visual display feedback are utilized.

RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.61/905,088, filed on Nov. 13, 2013.

This application is related to the following U.S. patent applications:

U.S. patent application Ser. No. 14/540,905, filed Nov. 13, 2014,entitled “Text Selection Using HMD Head-tracker and Voice-Command.”

U.S. patent application Ser. No. 14/540,939, filed Nov. 13, 2014,entitled “Head-Tracking Based Selection Technique For Head MountedDisplays (HMD).”

U.S. patent application Ser. No. 14/540,943, filed Nov. 13, 2014,entitled “Automatic Speech Recognition (ASR) Feedback For Head MountedDisplays (HMD).”

The entire teachings of the above applications are incorporated hereinby reference.

BACKGROUND OF THE INVENTION

Mobile computing devices, such as notebook PC's, smart phones, andtablet computing devices, are now common tools used for producing,analyzing, communicating, and consuming data in both business andpersonal life. Consumers continue to embrace a mobile digital lifestyleas the ease of access to digital information increases with high-speedwireless communications technologies becoming ubiquitous. Popular usesof mobile computing devices include displaying large amounts ofhigh-resolution computer graphics information and video content, oftenwirelessly streamed to the device. While these devices typically includea display screen, the preferred visual experience of a high-resolution,large format display cannot be easily replicated in such mobile devicesbecause the physical size of such device is limited to promote mobility.Another drawback of the aforementioned device types is that the userinterface is hands-dependent, typically requiring a user to enter dataor make selections using a keyboard (physical or virtual) ortouch-screen display. As a result, consumers are now seeking ahands-free high-quality, portable, color display solution to augment orreplace their hands-dependent mobile devices.

SUMMARY OF THE INVENTION

Recently developed micro-displays can provide large-format,high-resolution color pictures and streaming video in a very small formfactor. One application for such displays can be integrated into awireless headset computer worn on the head of the user with a displaywithin the field of view of the user, similar in format to eyeglasses,audio headset or video eyewear.

A “wireless computing headset” device, also referred to herein as aheadset computer (HSC) or head mounted display (HMD), includes one ormore small, high resolution micro-displays and associated optics tomagnify the image. The high resolution micro-displays can provide supervideo graphics array (SVGA) (800×600) resolution or extended graphicarrays (XGA) (1024×768) resolution, or higher resolutions known in theart.

A wireless computing headset contains one or more wireless computing andcommunication interfaces, enabling data and streaming video capability,and provides greater convenience and mobility through hands dependentdevices.

For more information concerning such devices, see co-pending patentapplications entitled “Mobile Wireless Display Software Platform forControlling Other Systems and Devices,” U.S. application Ser. No.12/348,648 filed Jan. 5, 2009, “Handheld Wireless Display Devices HavingHigh Resolution Display Suitable For Use as a Mobile Internet Device,”PCT International Application No. PCT/US09/38601 filed Mar. 27, 2009,and “Improved Headset Computer,” U.S. Application No. 61/638,419 filedApr. 25, 2012, each of which are incorporated herein by reference intheir entirety.

The described embodiments demonstrate how users can use natural orintuitive body movements or gestures to communicate with computers in amore natural way. A gesture may be a non-verbal communication of thebody that contains information. For example, waving one's hand mayconvey a message.

Head gestures are a natural and efficient way to respond to dialogboxes, especially when the user is already performing a different task.The movement in question must be a small, to prevent any discomfort tothe user, while being clearly decipherable by the system.

The present invention relates to gesture based interaction techniques tocontrol Head-Mounted Displays (HMD). In particular, embodiments providea head gesture-based recognition system where notification dialog boxescan be acknowledged by head nodding or ticking and question dialog boxescan be answered by head nods or head shakes.

In one aspect, the invention is a headset computer system, including aheadworn display unit. The headset computer system further includes adigital processor coupled to the display unit, and a head trackeroperatively coupled to the display unit and processor. The processor maybe configured to be responsive to any one of a set of predefined userhead gestures detected by the head tracker for making user selections.

In one embodiment, the predefined head gestures include one or more of atick movement, a head nodding movement, a head shaking movement, and ahead swiping movement. The head swiping movement may be one of a leftswiping movement and a right swiping movement.

In another embodiment, the display unit presents a notification dialogbox to the user, and the processor interprets a particular user headgesture as the user's acknowledgement of the notification dialog box.The particular user head gesture may a tick gesture. The processor mayinterpret a tick performed from left to right as having a first meaning,and a tick performed from right to left has having a second meaning,although the processor may interpret ticks performed in any and alldirections as having a common meaning.

In another embodiment, the display unit presents, to the user, aquestion dialog box having a first option and a second option. Theprocessor may interpret a first predetermined gesture as a selection ofthe first option and the processor interprets a second predeterminedgesture as a selection of the second option. In one embodiment, thefirst predetermined gesture is a head nod, and the second predeterminedgesture is a head shake. In another embodiment, the first predeterminegesture is a head swipe to the left or a head swipe to the right, andthe second predetermined gesture is a head swipe to the right or a headswipe to the left. In one embodiment, the head swipe to the left causesan indicator associated with the first option to transition from anempty state to a full state, and the head swipe to the right causes anindicator associated with the second option to transition from an emptystate to a full state.

Another embodiment includes an automatic speech recognition (ASR) systemconfigured to interpret utterances. The processor combines theinterpreted utterances with the detected gesture to produce a computerinterface input.

In another aspect, the invention is a method of making selections on aheadset computer system. The method includes presenting, by a processoron a display unit, an object to be selected. The method further includesdetecting, with a head tracker operatively coupled to the display unitand processor, a head gesture performed by a wearer of the headsetcomputer system. The head gesture may be associated with selection ofthe object. The method also includes selecting the object upon detectionof the head gesture.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

The foregoing will be apparent from the following more particulardescription of example embodiments of the invention, as illustrated inthe accompanying drawings in which like reference characters refer tothe same parts throughout the different views. The drawings are notnecessarily to scale, emphasis instead being placed upon illustratingembodiments of the present invention.

FIGS. 1A-1B are schematic illustrations of a headset computercooperating with a host computer (e.g., Smart Phone, laptop, etc.)according to principles of the present invention.

FIG. 2 is a block diagram of flow of data and control in the embodimentof FIGS. 1A-1B.

FIG. 3 is a schematic illustration of a tick head movement inembodiments.

FIG. 4 is a schematic illustration of a head nodding movement inembodiments.

FIGS. 5A-5C and 6A-6C are schematic views of a graphical user interfacein embodiments.

FIG. 7 is a flow diagram according to one embodiment of the invention.

DETAILED DESCRIPTION OF THE INVENTION

A description of example embodiments of the invention follows.

The teachings of all patents, published applications and referencescited herein are incorporated by reference in their entirety.

FIGS. 1A and 1B show an example embodiment of a wireless computingheadset device 100 (also referred to herein as a headset computer (HSC)or head mounted display (HMD)) that incorporates a high-resolution (VGAor better) microdisplay element 1010, and other features describedbelow.

HSC 100 can include audio input and/or output devices, including one ormore microphones, input and output speakers, geo-positional sensors(GPS), three to nine axis degrees of freedom orientation sensors,atmospheric sensors, health condition sensors, digital compass, pressuresensors, environmental sensors, energy sensors, acceleration sensors,position, attitude, motion, velocity and/or optical sensors, cameras(visible light, infrared, etc.), multiple wireless radios, auxiliarylighting, rangefinders, or the like and/or an array of sensors embeddedand/or integrated into the headset and/or attached to the device via oneor more peripheral ports 1020 (FIG. 1B).

Typically located within the housing of headset computing device 100 arevarious electronic circuits including, a microcomputer (single ormulticore processors), one or more wired and/or wireless communicationsinterfaces, memory or storage devices, various sensors and a peripheralmount or mount, such as a “hot shoe.”

Example embodiments of the HSC 100 can receive user input throughsensing voice commands, head movements, 110, 111, 112 and hand gestures113, or any combination thereof. A microphone (or microphones)operatively coupled to or integrated into the HSC 100 can be used tocapture speech commands, which are then digitized and processed usingautomatic speech recognition techniques. Gyroscopes, accelerometers, andother micro-electromechanical system sensors can be integrated into theHSC 100 and used to track the user's head movements 110, 111, 112 toprovide user input commands. Cameras or motion tracking sensors can beused to monitor a user's hand gestures 113 for user input commands. Sucha user interface may overcome the disadvantages of hands-dependentformats inherent in other mobile devices.

The HSC 100 can be used in various ways. It can be used as a peripheraldisplay for displaying video signals received and processed by a remotehost computing device 200 (shown in FIG. 1A). The host 200 may be, forexample, a notebook PC, smart phone, tablet device, or other computingdevice having less or greater computational complexity than the wirelesscomputing headset device 100, such as cloud-based network resources. Theheadset computing device 100 and host 200 can wirelessly communicate viaone or more wireless protocols, such as Bluetooth®, Wi-Fi, WiMAX, 4G LTEor other wireless radio link 150. (Bluetooth is a registered trademarkof Bluetooth Sig, Inc. of 5209 Lake Washington Boulevard, Kirkland,Wash. 98033).

In an example embodiment, the host 200 may be further connected to othernetworks, such as through a wireless connection to the Internet or othercloud-based network resources, so that the host 200 can act as awireless relay. Alternatively, some example embodiments of the HSC 100can wirelessly connect to the Internet and cloud-based network resourceswithout the use of a host wireless relay.

FIG. 1B is a perspective view showing some details of an exampleembodiment of a headset computer 100. The example embodiment HSC 100generally includes, a frame 1000, strap 1002, rear housing 1004, speaker1006, cantilever, or alternatively referred to as an arm or boom 1008with a built in microphone, and a micro-display subassembly 1010.

A head worn frame 1000 and strap 1002 are generally configured so that auser can wear the headset computer device 100 on the user's head. Ahousing 1004 is generally a low profile unit which houses theelectronics, such as the microprocessor, memory or other storage device,along with other associated circuitry. Speakers 1006 provide audiooutput to the user so that the user can hear information. Micro-displaysubassembly 1010 is used to render visual information to the user. It iscoupled to the arm 1008. The arm 1008 generally provides physicalsupport such that the micro-display subassembly is able to be positionedwithin the user's field of view 300 (FIG. 1A), preferably in front ofthe eye of the user or within its peripheral vision preferably slightlybelow or above the eye. Arm 1008 also provides the electrical or opticalconnections between the microdisplay subassembly 1010 and the controlcircuitry housed within housing unit 1004.

According to aspects that will be explained in more detail below, theHSC display device 100 allows a user to select a field of view 300within a much larger area defined by a virtual display 400. The user cantypically control the position, extent (e.g., X−Y or 3D range), and/ormagnification of the field of view 300.

While what is shown in FIGS. 1A and 1B is a monocular microdisplaypresenting a single fixed display element supported on the face of theuser with a cantilevered boom, it should be understood that othermechanical configurations for the remote control display device 100 arepossible.

FIG. 2 is a block diagram showing more detail of an embodiment of theHSC (or HMD) device 100, host 200 and the data that travels betweenthem. The HSC device 100 receives vocal input from the user via themicrophone, hand movements or body gestures via positional andorientation sensors, the camera or optical sensor(s), and head movementinputs via the head tracking circuitry such as 3 axis to 9 axis degreesof freedom orientational sensing. These are translated by software(processors) in the HSC device 100 into keyboard and/or mouse commandsthat are then sent over the Bluetooth or other wireless interface 150 tothe host 200. The host 200 then interprets these translated commands inaccordance with its own operating system/application software to performvarious functions. Among the commands is one to select a field of view300 within the virtual display 400 and return that selected screen datato the HSC device 100. Thus, it should be understood that a very largeformat virtual display area might be associated with applicationsoftware or an operating system running on the host 200. However, only aportion of that large virtual display area 400 within the field of view300 is returned to and actually displayed by the micro display 1010 ofHSC device 100.

The HSC 100 contains head-tracking capability. Head-tracking data iscaptured usually from an accelerometer as described above.

In one embodiment, the HSC 100 may take the form of the device describedin a co-pending US Patent Publication Number 2011/0187640, which ishereby incorporated by reference in its entirety.

Head gestures can be useful for interacting with human-machine interface(HMI) elements. An HMI sensitive to head gestures can lead to morenatural interaction between the user and the computer system.

Computer interfaces often interrupt a user's primary activity with anotification about an event or condition, which may or may not berelevant to the main activity.

In some systems, an HSC user must use their voice to acknowledge orattend to the notification before returning to the main activity. Forexample, the user might have to say “Confirm Selection” where an optionis available, to prevent the system from executing commands that theuser did not intend. This situation may arise when the Automatic SpeechRecognition (ASR) interprets an utterance from the user as a selectionor command that was not intended by the user.

The described embodiments enable the user to employ simple head gesturesas confirmations, declinations and other appropriate responsesassociated with certain actions. The gestures may take many forms, andwork in relation to a number of areas. One common use-case is nodding(i.e., head tilting to the front then to the back) or shaking the head(i.e., head turning to the left then to the right) to either confirm ordeny a confirmation request from the HSC system 100 (such as ‘Are yousure you wish to delete this document?).

The following descriptions outline two example embodiments fornotification dialog boxes, and question dialog boxes. The exampleembodiments outline a few example gestures.

Notification Dialog Boxes

Notification dialog boxes are one-button (single button) windows thatshow information from an application, and wait for the user toacknowledge the information.

Typically the notification dialog box contains one “OK” button, theselection of which allows the user to acknowledge the associatedinformation. There are a number of methods of confirming the informationand closing the box. For example, using head-tracking capability, theuser may manipulate a pointer to select the OK button. Alternatively,the HSC 100 may recognize a gesture, such as a particular head movement,as acknowledging the information.

In this example embodiment, the user traces a “tick” or “check” symbolwith the movement of his head to indicate acknowledgement. FIG. 3 showsan example of such a head-movement. In one embodiment, the checkmovement of the head may be detected by HSC 100 as an acknowledgementwhen the head motion occurs in either direction (i.e., right-to-left orleft-to-right). In other embodiments, the HSC 100 may interpret a checkmotioned in one direction as one gesture with first meaning, andinterpret a check motioned in the opposite direction as another gesturewith a second meaning (different from the meaning of the first gesture).

Question Dialog Boxes

Question dialog boxes are multiple button windows that display aquestion from the application and wait for positive or negative feedbackfrom the user. This type of dialog box includes both confirmation andrejection buttons. For example, a question dialog box may contain twobuttons—a “Yes” button and a “No” button.

Head nods and head shakes are a natural way in many cultures to signifypositive and negative feedback, respectively. A user, therefore, withlittle or no training can efficiently use a system that recognizes headnods and shakes. Answering a dialog box using a head gesture makes itpossible for users to keep focused on the task at hand.

FIG. 4 is illustrative of nodding action in embodiments. Additionalmovements may commonly accompany the basic movements shown in FIG. 4, soembodiments may account for those movements. For example, as some peoplemove their heads slightly upwards before the two steps shown in FIG. 4,some embodiments account for such extra steps in the specific iteration.

Two-Option Menus

Where two options, such as confirm or deny are shown to the user, oneoption is to use left/right head swiping.

Head Swipe Left—FIGS. 5A-5C illustrate example embodiments. When the HSC100 detects the user's head move to the left, the system 100 illustratesa button 460 on the left of the dialog box 450 filling with color.Embodiments next display a progress indicator 470 to the user to let himknow how long he needs to hover the pointer for the command (ofselectable object, e.g., button 460) to begin to execute.

The progress indicator 470 may be a circle or a vertical bar that fillswith color, transitions from dark to light, or from shaded to unshaded(or vice versa). Once the progress indicator 470 is fully colored (orshaded, etc.), the object or action (associated with the button 460) isselected (i.e., the selection of the command is consummated). In someembodiments, the system 100 may fill or completely color a progressindicator 470 in about two seconds. In other embodiments, the amount oftime necessary to completely fill the progress indicator may beselectable by the user (through, for example, a setup procedure of theHSC 100).

Head Swipe Right—As illustrated in FIGS. 6A-6C, the HSC 100 detects theuser's head movement to the right. System 100 illustrates the button 465on the right of the dialog box 450 filling with shading or color. System100 next displays to the user a progress indicator 470, as describedabove, to let him know how long he needs to maintain the pointer overbutton 465 for the corresponding command to execute.

The progress indicator 470 may be a circle or a vertical bar that fillswith color. Once the progress indicator 470 is full, the object oraction (associated with button 465) is selected. Typically system 100fills or completely colors the progress indicator 470 in about twoseconds.

Example Gesture Descriptions

Head Nodding—A nod of the head is a gesture in which the head is tiltedin alternating up and down arcs along the sagittal plane. In oneembodiment of an HSC 100 system, a single quick head nod is sufficientto indicate ‘Yes’.

Head Shaking—The Head Shake is a gesture in which the head is turnedleft and right along the transverse plane, horizontally, repeatedly inquick succession. In a HSC system 100 a two quick head shake movements,where the user's head turns to the left and then right, or vice versa,is sufficient to indicate ‘No’.

Head Ticking—The head moving diagonally from top left to bottom rightthen top right (or in reverse), in the shape of a tick (i.e., a checkmark).

Head Swipe Left/Right—The head is moved to the left or right and held inleft/right extended position to activate an option.

This is not an exhaustive list and this disclosure relates to gestureuse, utilizing the head-tracking capabilities generally.

Embodiments of this invention provide the user with an easy way toprovide input to a HSC 100 where voice-commands cannot be used or arenot the preference at that time.

FIG. 7 is a flow diagram according to one of the described embodiments.

It will be apparent that one or more embodiments described herein may beimplemented in many different forms of software and hardware. Softwarecode and/or specialized hardware used to implement embodiments describedherein is not limiting of the embodiments of the invention describedherein. Thus, the operation and behavior of embodiments are describedwithout reference to specific software code and/or specializedhardware—it being understood that one would be able to design softwareand/or hardware to implement the embodiments based on the descriptionherein.

Further, certain embodiments of the example embodiments described hereinmay be implemented as logic that performs one or more functions. Thislogic may be hardware-based, software-based, or a combination ofhardware-based and software-based. Some or all of the logic may bestored on one or more tangible, non-transitory, computer-readablestorage media and may include computer-executable instructions that maybe executed by a controller or processor. The computer-executableinstructions may include instructions that implement one or moreembodiments of the invention. The tangible, non-transitory,computer-readable storage media may be volatile or non-volatile and mayinclude, for example, flash memories, dynamic memories, removable disks,and non-removable disks.

While this invention has been particularly shown and described withreferences to example embodiments thereof, it will be understood bythose skilled in the art that various changes in form and details may bemade therein without departing from the scope of the inventionencompassed by the appended claims.

What is claimed is:
 1. A headset computer system, comprising: a headworndisplay unit; a digital processor coupled to the display unit; and ahead tracker operatively coupled to the display unit and processor, theprocessor configured to be responsive to a user head gesture detected bythe head tracker, and to manipulate a pointer presented on the displayunit, based on the detected user head gesture, to at least partiallyoverlap a selected object, the selected object being one of a firstobject that corresponds to a first option and a second object thatcorresponds to a second option; when the pointer overlaps the selectedobject: (i) the selected object gradually fills over a period of time,and (ii) a progress indicator is presented on the display in associationwith the selected object, and the progress indicator gradually fillsover a period of time, such that when the progress indicator iscompletely filled, a command associated with the selected object isconsummated.
 2. The system of claim 1, wherein the user head gesture isone of a left traced check symbol and a right traced check symbol. 3.The system of claim 2, wherein the processor interprets the check symbolperformed from left to right as having a first meaning, and the checksymbol performed from right to left has having a second meaning.
 4. Thesystem of claim 2, wherein the display unit presents, to the user, aquestion dialog box having a first option and a second option, theprocessor interprets a first predefined user head gesture as a selectionof the first option and the processor interprets a second predefineduser head gesture as a selection of the second option.
 5. The system ofclaim 4, wherein the first predefined user head gesture includes anadditional gesture that commonly accompanies the first predefined headgesture, which the processor interprets as a part of the firstpredetermined head gesture.
 6. The system of claim 5, wherein theadditional gesture includes one of a head swipe to the left or a headswipe to the right.
 7. The system of claim 4, wherein the head swipe tothe left causes an indicator associated with the first option totransition from an empty state to a full state, and wherein the headswipe to the right causes an indicator associated with the second optionto transition from an empty state to a full state.
 8. The system ofclaim 1, wherein the period of time is selectable by a user of theheadset computer system through a setup procedure of the headsetcomputer system.
 9. The system of claim 1, further including anautomatic speech recognition (ASR) system configured to interpretutterances, wherein the processor combines the interpreted utteranceswith the detected gesture to produce a computer interface input.
 10. Amethod of making selections on a headset computer system, comprising:presenting, by a processor on a display unit, a first object thatcorresponds to a first option and a second object that corresponds to asecond option; detecting, with a head tracker operatively coupled to thedisplay unit and processor, a head gesture performed by a wearer of theheadset computer system; manipulating a pointer presented on the displayunit, based on the detected user head gesture, to at least partiallyoverlap a selected object, the selected object being one of a firstobject that corresponds to a first option and a second object thatcorresponds to a second option; and when the pointer overlaps theselected object: (i) filling the selected object gradually over a periodof time, and (ii) presenting a progress indicator on the display inassociation with the selected object, and filling the progress indicatorgradually over a period of time; and (iii) consummating a commandassociated with the selected object when the progress indicator iscompletely filled.
 11. The method of claim 10, wherein the head gestureis one of a left traced check symbol and a right traced check symbol.12. The method of claim 11, further including interpreting the checksymbol performed from left to right as having a first meaning, and ainterpreting the check symbol performed from right to left as having asecond meaning.
 13. The method of claim 11, further includingpresenting, to the user, a question dialog box having a first option anda second option, interpreting a first head gesture as a selection of thefirst option and interpreting a second head gesture as a selection ofthe second option.
 14. The method of claim 13, wherein the first headgesture includes an additional gesture that commonly accompanies thefirst head gesture, which the processor interprets as a part of thefirst head gesture.
 15. The method of claim 14, wherein the additionalgesture includes one of a head swipe to the left or a head swipe to theright.
 16. The method of claim 10, further including adjusting theperiod of time based on an input provided by a user of the headsetcomputer system through a setup procedure of the headset computersystem.